The Effects of Scaling on Trends of Development: Classical Test Theory and Item Response Theory

The scale metrics used in educational testing are often arbitrary, and this can impact interpretation of scores on measurements. Both classical test theory sum scores and item response theory estimates measure the same underlying dimension, but differences in the two scales may lead one to be more preferential than the other in interpreting data. Mismatch between individual ability and test difficulty can further result in difficulties in correctly interpreting trends of development in longitudinal data. A previous limited simulation by Embretson (2007) demonstrated that classical test theory sum scores result in misinterpretation of linear trends of development, and that item response theory estimates improve upon the problem. This study replicates the results from the previous literature, as well as extends the results to include simulation of development in both quadratic and cubic trends. Results indicate that while item response theory scaling does improve estimates for the linear, quadratic, and cubic trends simulated, ultimately the two methods perform very similar to one another. Item response theory estimates resulted in marginally fewer Type I and Type II errors, especially when investigating interaction effects. The mismatch between test difficulty and ability level of test takers has the strongest impact on correctly interpreting how individuals develop over time. iv Acknowledgements I would like to express gratitude to my advisor Dr. Bovaird for his help in completing my master thesis, as well as for introducing me to my topic. I would also like to thank my reader Dr. De Ayala for his course on IRT that helped me learn how to do the simulation necessary for my thesis. I would like to thank my family, Zane, Wendy, and Alyssa, for always being there. Your love and warmth has meant the world to me and always helps me get through any problems I face. Thank you to all my extended family members, who are too numerous to name here but all of whom have had a lasting impact on my life. All of you never stopped believing in me, and that has always kept me going. Special thanks to my friends here at UNL, including HyeSun Lee, the rest of the birthday club, and the guys that manned the NEAR Center with me. I could not have made it through without all of you making my graduate career fun. To my friends back in Blanco and the surrounding areas, thanks for always believing in me. Last …